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The Institute for Higher Education Policy reviewed the 
research on quality and equality in Internet-based higher education and found 
a relative paucity of original research dedicated to explaining or predicting 
phenomena related to distance learning. ' The research that does exist has 
tended to emphasize student outcomes for individual courses, rather than a 
whole academic program. The research does not address the question of dropout 
rates or how learning styles relate to the use of particular technologies. 

The research does not include a theoretical or conceptual framework, nor does 
it consider the effectiveness of "digital libraries." The Institute developed 
benchmarks for Internet-based distance learning drawing on the limited 
research available and a case study that consisted of three sequential 
phases: a literature review, the identification of institutions with 
experience in Internet-based distance education, and a survey of six such 
institutions. The outcome was a list of 24 benchmarks, attached to this 
document as an appendix. The development of this list of benchmarks has 
resulted in the formulation of three recommendations for Internet-based 
distance education: (1) proceed cautiously with efforts to- apply a 

governmental imprimatur on Web-based higher education; (2) find ways to 
strengthen the impact technology may have on equal educational opportunity, 
especially institutions that serve large numbers of underrepresented 
students; and (3) ground government policies in standards of quality that are 
based on what works, such as the proposed benchmarks. (SLD) 
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Distinguished Colleagues: 



I am delighted to be here with you on this important occasion. For those of you who may 
not know our work, the Institute for Higher Education Policy is an independent policy 
research and consulting NGO whose main focus is access and opportunity in higher 
education. This work has been primarily in the context of U.S. higher education, though 
increasingly the Institute has had the opportunity to work on governmental and institutional 
policy development in several nations, including South Africa, Mozambique, Russia, 
Kyrgyzstan, and others. We also are proud contributors to policy development here in 
Puerto Rico, having served as an analytic resource to the Puerto Rico Council on Higher 
Education for several years. 

The Institute was founded in 1993 to address what continues to be a fundamental paradox 
of American higher education; namely, that as participation in higher education has 
increased in the last 30 years, the overall status of historically disadvantaged populations 
has not significantly improved, and may even have gotten worse. Today, nearly 70% of 
high school graduates go on to college, but this very impressive statistic masks the fact that 
the college participation gap between low- and high- income students, and between whites 
and other racial and ethnic groups in the U.S., has remained almost exactly what it was in 
the early 1 970s — 30 percentage points. Since non- whites are the fastest growing segment 
of the American population, and given the growing disparities between low- and high- 
income populations, the failure to close that gap means that low- income individuals and 
people of color are actually losing ground, a fact that has very real consequences for our 
overall economic and social growth and stability. 

The Institute publishes more than a dozen reports per year on a range of issues, including 
financing of higher education, quality assurance, demographic trends, minority- serving 
colleges and universities, and other issues. Technology, and its overall impact on 
educational opportunity and quality, has been one of our foremost concerns. As the 
convening of this important meeting attests, technology is affecting higher education on 
many levels — ranging from scientific equipment to financing structures to university 
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administrative systems. But the place where technology is likely to have the biggest impact 
over the long-term is in the teaching and learning context. Today, one half of all college 
courses in the U.S. use the Internet in some way as a part of the educational process, 
including class web pages, e-mail discussion groups, etc., and nearly one out of every 10 
American college students takes at least one distance education course in the course of 
obtaining their degree — particularly one- and two-way interactive video, and online 
courses, which is where most of the growth is now taking place. The Institute’s work in 
this area is focused on understanding how technology is changing postsecondary education, 
and what that means for student equity and overall standards of quality. My perspectives 
on issues related to the quality and equality of Internet-based higher education are largely 
drawn from my experiences with this method of higher education delivery in the U.S., 
though I am convinced from our work in other nations that many of the principles and 
findings that I will discuss today are applicable in many national and cross-national 
contexts. 

Two of our recent studies have been discussed widely in the higher education community in 
the last two years, and provide a useful starting point for my remarks. The first, entitled 
What’s the Difference ? A Review of Contemporary Research on the Effectiveness of 
Distance Learning in Higher Education, was intended to shed some light on the question of 
the differences in educational outcomes between distance learning and traditional 
classroom-based instruction. (A copy of the full report can be found on the Institute’s 
website at www.ihep.org .) In our prior work in U.S. higher education policy, we had 
observed that a growing number of policymakers (particularly in states where higher 
education enrollments are projected to surge in the next decade, such as in the West) were 
pronouncing the end of new campus construction in favor of distance-based instruction. 
These pronouncements seemed to be based primarily on the belief that the outcomes of 
distance-based and traditional classroom-based instruction are indistinguishable — that there 
is no significant difference between the outcomes of distance versus traditional classroom- 
based learning. 
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To help answer this question, we decided to assess the available evidence by reviewing 
what has been published about the issue in recent years. Because of the constantly evolving 
nature of the technology and its uses, we limited the scope of our review to material 
published during the 1990s. This still left us with the task of reviewing several hundred 
articles, papers, and dissertations. It turned out that most of what is written about distance 
learning s opinion pieces, how-to articles, and second-hand reports that do not involve 
actual research on the students or faculty involved with distance learning. We therefore 
focused our inquiry only on those items and case studies that contained original research, in 
the hope that they would provide enough evidence to assess the differences between 
distance and classroom-based teaching. What we found was that despite the large volume 
of written material concentrating on distance learning, there is a relative paucity of original 
research dedicated to explaining or predicting phenomena related to distance learning. 

Our critique of the quality of the research made four simple points. First, much of the 
research does not control for extraneous variables and therefore cannot show cause and 
effect. Second, most of the studies do not use randomly selected subjects; many of the 
published studies reviewed used intact groups (such as an entire class of students) for 
comparison purposes. Third, the validity and reliability of the instruments used to measure 
student outcomes and attitudes, such as questionnaires and surveys, are questionable. And 
fourth, many studies do not adequately control for the feelings and attitudes of the students 
and faculty — what the educational research refers to as “reactive effects” — which can 
sometimes skew the findings by providing short-term or temporary benefits that are not 
sustained over the span of the educational program. 

In addition to these general weaknesses in the quality of the research, the report found that 
the research did not address several significant issues regarding the effectiveness of 
distance learning. These gaps must be filled so that the public policy discussions about 
distance education can be based on accurate and adequate information. Examples of these 
gaps include: 
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The research has tended to emphasize student outcomes for individual courses rather 
than for a total academic program. 

A major gap in the research is the lack of studies dedicated to measuring the effectiveness 
of total academic programs taught using distance learning. This raises serious questions 
about whether a total academic program delivered by technology compares favorably with a 
program provided on-campus — an important distinction given that government policies and 
programs, such as student aid, typically are aimed at providing access to degrees or a 
program of study, not just single courses. 

The research does not adequately explain why the drop-out rates of distance learners are 
higher. 

In a number of studies, there was evidence that a higher percentage of students participating 
in a distance learning course tended to drop out before the course was completed compared 
to students in a conventional classroom. The issue of student persistence is troubling 
because of both the negative consequences associated with dropping out, and the fact that 
the research could be excluding these dropouts — thereby tilting the student outcome 
findings toward those who are “successful.” 

The research does not take into consideration how the different learning styles of 
students relate to the use of particular technologies. 

Our understanding of how the learner, the learning task, and a particular technology interact 
is limited. Learner characteristics, such as gender, age, and educational experience, are a 
major factor in the achievement and satisfaction levels of the distance learner. Information 
regarding a student’s preferred learning style — for example, how he or she processes 
information — will influence how the course is designed and what type of technology is 
used. Additional research could result in more information about why different 
technologies might be better suited for specific learning tasks. 

The research does not include a theoretical or conceptual framework. 

There is a vital need to develop a more integrated, coherent, and sophisticated program of 
research on distance learning that is based on theory. Theory allows researchers to build on 
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the work of others and, therefore, increase the probability of addressing the more significant 
questions regarding distance learning. 

The research does not adequately address the effectiveness of digital “ libraries . ” 

Students participating in distance learning, particularly those in remote locations, are often 
introduced to a digital “library” that provides access to bibliographies and full-text of a 
variety of resources. But do digital libraries provide adequate services for the academic 
programs they are established to support? Anecdotal evidence seems to suggest that the 
curriculum objectives of some distance learning courses have been altered because of a 
limited variety of books, journals, and other resources available from the digital library. 

The report generated considerable dialogue throughout academia about what constitutes 
quality in distance learning settings. It also pointed out areas for more research, including 
how, and in what ways, technology can enhance the teaching/leaming process. Last year, 
the Institute decided to follow-up What’s the Difference? with another study. Given the 
first report’s major conclusions, we decided to contribute to the research by looking at some 
fundamental benchmarks of quality in Internet-based distance education programs by 
conducting case studies of actual programs. We recognized that such a study inherently 
would include some of the very limitations noted in What’s the Difference, but we were 
convinced that such research can help build a foundation for future analyses capable of 
refining or expanding upon the lessons learned from the institutions studied. That study, 
entitled, Quality On the Line: Benchmarks for Success in Internet-based Distance 
Education, has generated a flurry of interest from higher education leaders in many 
countries. In fact, Quality On the Line has been downloaded more than 100,000 times from 
the Institute’s website since its release last year — a testament to the enormous interest that 
this topic now generates across the globe. 

The extraordinary growth of technology- mediated distance learning in higher education has 
prompted several different U.S. -based organizations to develop principles, guidelines, or 
benchmarks to ensure quality distance education. These organizations include the 
American Council on Education, the Global Alliance for Transnational Education, the 
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National Education Association, and the Southern Regional Electronic Campus, among 
others. The quality assurance benchmarks promoted by these organizations are designed to 
apply to a wide variety of institutional contexts and consist of fairly broad statements. 
Virtually all of the strategies address such topics as course development, faculty training, 
student services, learning resources, infrastructure, and outcomes assessment. 

These benchmarks for all types of distance learning have been in existence in various forms 
for a number of years. The question that has arisen is whether they are applicable to 
Internet -based distance education. In short, are the benchmarks appropriate and necessary 
to ensure quality Internet-based distance education? Two organizations — the National 
Education Association and Blackboard, Inc., a widely used platform provider for online 
distance education— jointly commissioned the Institute to examine the benchmarks by 
studying active distance learning programs at several institutions. 

Specifically, the project sponsors asked the Institute to attempt to validate those 
benchmarks that have been published by various entities, with specific attention to 
Internet-based distance education. The study was designed to ascertain the degree to which 
the benchmarks are actually incorporated in the policies, procedures, and practices of 
colleges and universities that are distance education leaders. In addition, this case study 
sought to determine how important the benchmarks are to the institutions, including faculty, 
administrators, and students. 

The case study process consisted of three sequential phases. First, a comprehensive 
literature search was conducted to compile those benchmarks recommended by other 
organizations and groups, as well as those suggested in various articles and publications. 
This search resulted in a total of 45 benchmarks developed by these other organizations. 

Second, institutions were identified that satisfied the following criteria. The institutions 
must (1) have substantial experience in distance education; (2) be recognized as among the 
leaders in Internet-based distance education; (3) be regionally accredited; and (4) offer 
more than one degree program via online distance learning. To ensure that a broad 
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spectrum of higher education institutions were represented, the study included a community 
college, a comprehensive institution, a research institution, and a virtual institution. From 
among several colleges and universities that fit the requirements, the following six 
institutions agreed to participate in the study: Brevard Community College in Florida, 
Regents College in New York, the University of Illinois at Urbana-Champaign, the 
University of Maryland University College, Utah State University, and Weber State 
University in Utah. 

Third, the six institutions were visited by Institute staff to assess the degree to which the 
campuses incorporated the benchmarks in their Internet-based distance learning courses and 
programs. Each site visit included interviews with faculty, administrators, and students, as 
well as a survey of these individuals that rated both the presence and importance of the 
original group of 45 benchmarks to determine if they are being followed and if they make a 
difference in terms of academic quality. 

The results of the study revealed that, for the most part, the benchmarks for quality 
Internet-based distance education were considered important and, in general, the institutions 
strove to incorporate them into their policies, practices, and procedures. At the same time, 
several benchmarks did not enjoy consensus among administrators, faculty, and students at 
the institutions and, in some instances, were not considered mandatory to ensure quality in 
distance education. 

The final outcome was a list of 24 benchmarks that are essential to ensure quality in 
Internet-based distance education (see Appendix). Stated differently, the absence of the 
benchmark would be deleterious to quality. The purpose of the study was to assist 
policymakers — college and university presidents and chief academic officers, state 
coordinating boards, accrediting bodies, state legislatures, and governors’ offices — as well 
as faculty and students, make reasonable judgments with regard to quality Internet-based 
distance education. The challenge, then, was to identify those benchmarks that are essential 
for quality distance education — in contrast to those benchmarks that contribute to and 
support the teaching/leaming process, but are not necessary or required to ensure quality. 
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While we certainly don’t believe this is a definitive list, we are confident that policymakers 
can use this list with the assurance that they are directly addressing the issue of quality 
without placing unnecessary restrictions on institutions of higher education. 

Recommendations 

The completion of this second study has in turn highlighted other issues that merit further 
consideration, including how to define student success in distance-based education and 
what measures should be used. We also have begun to explore the important issues related 
to financing technology in higher education, and earlier this year published a primer on the 
topic entitled Funding the Infostructure. We hope to continue to explore these and other 
questions in the next few years. 

As you move ahead with your important work as global leaders in higher education and 
information technology, I hope you will take into account the lessons we and other analysts 
have learned from research regarding Internet-based higher education in the United States. 
Specifically, I would urge you to consider the following three recommendations, all 
grounded in our work: 

1) Proceed cautiously with efforts to apply a governmental imprimatur on web-based 
higher education. 

Web-based education holds great promise as a tool for improving both the scope and 
effectiveness of higher education. But as with other educational innovations that have 
come before it, there is some danger that the innovations made possible through the 
Internet are advancing more rapidly than our understanding of its practical uses. 
Princeton historian Robert Damton makes this point in an essay about electronic 
publishing published in the March 18, 1999 issue of the New York Review of Books. 
Damton observes that, since its inception, electronic publishing has passed through 
three stages: “an initial phase of utopian enthusiasm, a period of disillusionment, and a 
new tendency toward pragmatism.” In the context of higher education, more emphasis 
has been placed on the “utopian” possibilities of the technology and its potential to 
transform teaching and learning. But not enough “pragmatism” has been applied to 
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allow for a discussion of technology’s practical implications — and limitations — as a 
supplement to enhance teaching and learning. Getting at those pragmatic and ultimately 
crucial questions is what all those with an interest in higher education’s future should 
make as their common goal. 

I would be particularly concerned about attempts to make dramatic changes to funding 
systems in the absence of reliable and definitive information about how such resources 
would be used, and for what purposes. A good example is an area that I have spent a 
great deal of my time studying professionally — student financial aid. Internet-based 
distance learning certainly will challenge our concepts and assumptions in student aid 
delivery. But it is critical to recognize that distance learning and traditional classroom 
based instruction are integrally linked, and not wholly disparate in their goals or 
techniques. While the methods of teaching and learning that are possible as a result of 
the Internet certainly have the capacity to transform our educational systems and 
processes, we should not view them as separate from traditional higher education. On 
the contrary, what happens in one should be viewed as having very real implications for 
the other. 

Focusing on ways to enhance student aid to assure that its principle goal of providing 
access to college is a worthy endeavor, especially for public policy. Yet in so doing, we 
must be certain that our decisions are based on the desire to make college possible for 
all who have the interest and skills, and not to favor any particular mode of delivery. 
Access to quality higher education opportunities must continue to be the most important 
and enduring consideration of student aid systems. If the Internet can offer greater focus 
on that goal, then it certainly could help to bridge the gap in educational opportunities 
still present in many societies. But bending the system to favor any one particular mode 
of delivery will only further erode the essential purposes for which need-based aid 
programs were created. We must not allow our enthusiasm for, or fear of, the 
technology to impede progress in making the goal of equal educational opportunity a 
reality. 
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2) Find ways to strengthen the impact that technology may have on equal 
educational opportunity, and particularly on institutions of higher education that 
serve large numbers of underrepresented students. 

Numerous reports and studies in the U.S., Africa, Europe, and other parts of the world 
have observed that the demographic compositions of the technologically 
disenfranchised are overwhelmingly ethnic minorities, the poor, and residents of non- 
urban areas. Combined with other indices relative to these populations, such as 
unemployment and adult illiteracy, these groups will continue to be alienated from the 
technology resources of the public and private sectors unless interventions are 
introduced. 

Some efforts are underway to help build the technology infrastructure to serve 
disadvantaged communities, ranging from the important accomplishments of the Maori 
universities in New Zealand to the groundbreaking work of the African Virtual 
University. In the U.S., modest investments have been made in addressing the 

technology deficits that exist at Minority- Serving Institutions (MSIs) of higher 
education — Historically and Predominantly Black Colleges and Universities (HBCUs), 
Hispanic- Serving Institutions (HSIs), and Tribal Colleges and Universities. For 
example, the National Science Foundation is investing $6 million over four years to 
improve the Internet connections at a select group of these minority- serving institutions. 
Recently proposed national legislation, as yet unfunded, would add another $250 
million in resources to support the technology infrastructure at these institutions. 

However, providing access to the Internet is a necessary but not sufficient component of 
making institutions that serve disadvantaged populations viable players in the online 
learning community. Maximizing technology’s effectiveness requires well- trained and 
supported faculty who can use the Internet as a major component of the teaching and 
learning process. Government policy needs to be certain that these institutions, and the 
students they serve, are not left behind by the Internet explosion. Making sufficient 
resources available for how to use technology is equally as important as investment in 
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the technology infrastructure. Specific funding could be targeted at faculty training for 
teaching online, as well as broader training efforts for students and faculty in web-based 
teaching and research methods. 

3) Ground government policies in standards of quality that are based on what 
works — such as the benchmarks that have been proposed. 

This really requires no further explanation beyond what I have described, except to 
emphasize that access to poor quality higher education really is not equalizing 
educational opportunity. Given the historic and continuing role of governments in 
fostering equal opportunity, it is critical to ensure that such access is to the highest 
quality programs possible — irrespective of whether it is classroom-based, Internet- 
based, or any other type of delivery mechanism. 

In closing, I would observe that the work that we have engaged in at this meeting is 
significant, timely, and ultimately could have long-term implications for the important role 
that higher education plays in the economic and social growth and stability of nations 
across the globe. Thank you very much for the opportunity to be with you at this important 
conference, and to share these thoughts on the implications of Internet-based higher 
education on educational quality and equality. 
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APPENDIX 



Benchmarks for Success in Internet-based Distance Education 



Institutional Support Benchmarks 

$ A documented technology plan that includes electronic security measures (i.e., 
password protection, encryption, back-up systems) is in place and operational to 
ensure both quality standards and the integrity and validity of information. 

$ The reliability of the technology delivery system is as failsafe as possible. 

$ A centralized system provides support for building and maintaining the distance 

education infrastructure. 

Course Development Benchmarks 

$ Guidelines regarding minimum standards are used for course development, design, 
and delivery, while learning outcomes — not the availability of existing 
technology — determine the technology being used to deliver course content. 

$ Instructional materials are reviewed periodically to ensure they meet program 
standards. 

$ Courses are designed to require students to engage themselves in analysis, synthesis, 
and evaluation as part of their course and program requirements. 

Teaching/Learning Benchmarks 

$ Student interaction with faculty and other students is an essential characteristic and 
is facilitated through a variety of ways, including voice- mail and/or e-mail. 

$ Feedback to student assignments and questions is constructive and provided in a 
timely manner. 

$ Students are instructed in the proper methods of effective research, including 
assessment of the validity of resources. 

Course Structure Benchmarks 

$ Before starting an online program, students are advised about the program to 
determine (1) if they possess the self-motivation and commitment to learn at a 
distance and (2) if they have access to the minimal technology required by the 
course design. 
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$ Students are provided with supplemental course information that outlines course 
objectives, concepts, and ideas, and learning outcomes for each course are 
summarized in a clearly written, straightforward statement. 

$ Students have access to sufficient library resources that may include a “virtual 
library” accessible through the World Wide Web. 

$ Faculty and students agree upon expectations regarding times for student 
assignment completion and faculty response. 

Student Support Benchmarks 

$ Students receive information about programs, including admission requirements, 
tuition and fees, books and supplies, technical and proctoring requirements, and 
student support services. 

$ Students are provided with hands-on training and information to aid them in 
securing material through electronic databases, interlibrary loans, government 
archives, news services, and other sources. 

$ Throughout the duration of the course/program, students have access to technical 
assistance, including detailed instructions regarding the electronic media used, 
practice sessions prior to the beginning of the course, and convenient access to 
technical support staff. 

$ Questions directed to student service personnel are answered accurately and 
quickly, with a structured system in place to address student complaints. 

Faculty Support Benchmarks 

$ Technical assistance in course development is available to faculty, who are 
encouraged to use it. 

$ Faculty members are assisted in the transition from classroom teaching to online 
instruction and are assessed during the process. 

$ Instructor training and assistance, including peer mentoring, continues through the 
progression of the online course. 

$ Faculty members are provided with written resources to deal with issues arising 
from student use of electronically- accessed data. 
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Evaluation and Assessment Benchmarks 



$ The program’s educational effectiveness and teaching/leaming process is assessed 
through an evaluation process that uses several methods and applies specific 
standards. 

$ Data on enrollment, costs, and successful/innovative uses of technology are used to 
evaluate program effectiveness. 

$ Intended learning outcomes are reviewed regularly to ensure clarity, utility, and 
appropriateness. 
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